Skip to content

fix(cuda): support Turing GPUs - #24

Open
lukasrakauskas wants to merge 1 commit into
FlashML-org:mainfrom
lukasrakauskas:fix/sm75-turing
Open

fix(cuda): support Turing GPUs#24
lukasrakauskas wants to merge 1 commit into
FlashML-org:mainfrom
lukasrakauskas:fix/sm75-turing

Conversation

@lukasrakauskas

@lukasrakauskas lukasrakauskas commented Aug 22, 2026

Copy link
Copy Markdown

Adds NVIDIA Turing (sm_75) support for RTX 20-series GPUs:

  • Disable optional kernels that require sm_80+.
  • Use Turing-safe attention tile sizes.
  • Add sm_75 to kernel-cache builds.
  • Add backend and attention tests.

Verified on RTX 2060 SUPER with the Qwen3.6 NVFP4 model. The API server starts and serves
requests successfully.

The full test suite ran with 1,313 passed, 25 skipped, and 39 GPU-specific failures caused by RTX 2060 SUPER (sm_75) limitations and unavailable optional backends.

@tuxevil

tuxevil commented Aug 29, 2026

Copy link
Copy Markdown

Independent sm_75 validation from a Quadro RTX 4000 (Turing, 8 GB).

I've been testing #131 (feat/generic-gguf) with Qwen3.6-35B-A3B on:

  • NVIDIA Quadro RTX 4000, sm_75, 8 GB
  • Ryzen 9 7945HX
  • 96 GB DDR5
  • PyTorch 2.11 / CUDA 13.3

The _supports_sm80() capability gating from this PR turned out to be necessary when combining the new GGUF path with Turing.

Without the gating, sgl_kernel was importable and therefore selected, but failed later at CUDA launch with:

no kernel image is available for execution on the device

After applying the backend capability gates from #24 to the #131 branch, the server correctly fell back to Turing-compatible paths and I was able to run prolonged Qwen3.6 benchmarks successfully.

Models exercised successfully on sm_75 included:

  • Qwen3.6-35B-A3B IQ3_S GGUF
  • Qwen3.6-35B-A3B Q4_K GGUF
  • Qwen3.6-35B-A3B Q5_K_S GGUF
  • native Qwen3.6 NVFP4 using the Triton NVFP4 backend

The most useful observation from my side is that checking whether these optional packages are merely importable is not sufficient on Turing: they can import successfully while containing no runnable sm_75 kernel, producing a late runtime failure instead of selecting the fallback.

So the architecture check in this PR fixes a real failure mode I could independently reproduce on another Turing card.

For clarity: I specifically transplanted/tested the optional-backend sm80 gating from this PR onto #131. I did not independently validate every other change in #24 (e.g. the kernel-cache packaging changes), so I don't want to overstate the scope of this report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants